Papers by Jordan Lee Boyd-Graber
AUDITA: A New Dataset to Audit Humans vs. AI Skill at Audio QA (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing audio question answering benchmarks emphasize sound event classification or caption-grounded queries. |
| Approach: | They propose a large-scale, real-world audio question answering benchmark to evaluate audio reasoning beyond surface-level acoustic recognition. |
| Outcome: | The proposed model achieves 32.13% accuracy while demonstrating comprehension of audio . state-of-the-art models perform poorly, with average accuracy below 8.86%. |
Large Language Models Struggle to Describe the Haystack without Human Help: A Social Science-Inspired Evaluation of Topic Models (2025.acl-long)
Copied to clipboard
Zongxia Li, Lorena Calvo-Bartolomé, Alexander Miserlis Hoyle, Paiheng Xu, Daniel Kofi Stephens, Juan Francisco Fung, Alden Dima, Jordan Lee Boyd-Graber
| Challenge: | a common use of NLP is to facilitate the understanding of large document collections. |
| Approach: | They propose to use large language models to replace probabilistic topic models in real-world applications. |
| Outcome: | The proposed model generates more human-readable topics and shows higher average win probabilities than traditional models for data exploration. |
Reverse Question Answering: Can an LLM Write a Question so Hard (or Bad) that it Can’t Answer? (2025.naacl-short)
Copied to clipboard
| Challenge: | Question answering (QA) is a popular task, but we test both separately . a recent study found that LLMs are less accurate in numerical RQA than RQA . |
| Approach: | We run 16 LLMs on QA and RQA with trivia questions/answers . they find question and answer types that lead to RQA errors and suggest improvements . |
| Outcome: | The results show that LLMs are less accurate in RQA for numerical answers than RQA . RQA errors correlate with question difficulty and inversely correlate with answer frequencies . |
Whose Boat Does it Float? Improving Personalization in Preference Tuning via Inferred User Personas (2025.acl-long)
Copied to clipboard
| Challenge: | a recent study shows that LLMs can't tailor outputs to users with uncommon preferences . despite the success of persona inference, we may need debiasing and abstention. |
| Approach: | They propose to use preference data to infer needs and interests of users who prefer either output . they argue that training on preference data augmented with PI boosts personalization . |
| Outcome: | The proposed method can be used to improve personalization with less privacy concerns. |
Measuring User’s Mental Models of Speech Translation in Human-AI Collaboration (2026.acl-long)
Copied to clipboard
| Challenge: | Existing research on machine translation tools has not revealed how users perceive MT errors and how they evolve through interaction. |
| Approach: | They propose a framework where users accept MT output or request professional re-translation to answer questions based on information presented in a foreign language. |
| Outcome: | The proposed framework can predict where the system is likely to be wrong and how it evolves through interaction. |
Should I Trust You? Detecting Deception in Negotiations using Counterfactual RL (2025.findings-acl)
Copied to clipboard
Wichayaporn Wongkamjan, Yanze Wang, Feng Gu, Denis Peskoff, Jonathan K. Kummerfeld, Jonathan May, Jordan Lee Boyd-Graber
| Challenge: | Future human-AI interaction tools can build on our methods for deception detection by triggering friction to give users a chance to interrogate suspicious proposals. |
| Approach: | They propose to use CTRL-D to detect deception in a board game called Diplomacy . CTRL is a counterfactual RL that has a good recall and almost perfect precision . future tools could build on this to reevaluate trust in suspicious negotiations . |
| Outcome: | The proposed method detects human deception with a high precision when compared to a Large Language Model approach that flags many true messages as deceptive. |
No Questions are Stupid, but some are Poorly Posed: Understanding Poorly-Posed Information-Seeking Questions (2025.acl-long)
Copied to clipboard
| Challenge: | When a question is poorly posed, answerers struggle to converge on dominant interpretations, while models attempt comprehensive coverage by addressing many interpretations simultaneously. |
| Approach: | They propose a computational framework to study poorly-posedness of questions by generating spaces of potential interpretations and computing distributions based on interpretations chosen by answerers in the Reddit question thread. |
| Outcome: | The proposed framework analyzes poorly-posed questions using a set of interpretations chosen by human answerers and large language models. |
Discrepancy Detection at the Data Level: Toward Consistent Multilingual Question Answering (2025.emnlp-main)
Copied to clipboard
Lorena Calvo-Bartolomé, Valérie Aldana, Karla Cantarero, Alonso Madroñal de Mesa, Jerónimo Arenas-García, Jordan Lee Boyd-Graber
| Challenge: | Multilingual question answering systems must ensure factual consistency across languages while also accounting for cultural variation in subjective responses. |
| Approach: | They propose a user-in-the-loop fact-checking pipeline to detect factual and cultural discrepancies in multilingual QA knowledge bases. |
| Outcome: | The proposed tool detects factual and cultural discrepancies in bilingual question answering systems. |
Which of These Best Describes Multiple Choice Evaluation with LLMs? A) Forced B) Flawed C) Fixable D) All of the Above (2025.acl-long)
Copied to clipboard
| Challenge: | Multiple choice question answering (MCQA) is popular for LLM evaluation due to its simplicity and human-like testing. |
| Approach: | They argue for a reform of multiple choice question answering (MCQA) they argue for more generative formats based on human testing . |
| Outcome: | The proposed reforms improve the quality of MCQA, the authors argue . they show that even when MCQ is a useful format, its datasets suffer from leakage, unanswerability, shortcuts and saturation. |
ProxAnn: Use-Oriented Evaluations of Topic Models and Document Clustering (2025.acl-long)
Copied to clipboard
| Challenge: | Topic models and document clustering evaluations often use automated metrics that align poorly with human preferences or require expert labels that are intractable to scale. |
| Approach: | They propose a protocol for evaluating topic models and document clustering evaluations that uses crowdworker annotations to validate automated proxies. |
| Outcome: | The proposed protocol is scalable and easy to adapt to an LLM prompt. |
Is your benchmark truly adversarial? AdvScore: Evaluating Human-Grounded Adversarialness (2025.naacl-long)
Copied to clipboard
| Challenge: | As models evolve, datasets can become outdated. |
| Approach: | They propose a human-grounded evaluation metric that assesses adversarialness by capturing models’ and humans’ varying abilities, while also identifying poor examples. |
| Outcome: | The proposed evaluation metric measures the accuracy of an adversarial question answering dataset and determines whether models are performing well on the dataset. |
Language Models Don’t Know What You Want: Evaluating Personalization in Deep Research Needs Real Users (2026.acl-long)
Copied to clipboard
Nishant Balepur, Malachi Hamada, Varsha Kishore, Sergey Feldman, Amanpreet Singh, Pao Siangliulue, Joseph Chee Chang, Eunsol Choi, Jordan Lee Boyd-Graber, Aakanksha Naik
| Challenge: | Earlier research used real users to push personalization, but easy-to-use judges have been criticized for not adopting online studies. |
| Approach: | They propose a personalized action-following tool that infers a user's research interests and proposes personalized actions for a query. |
| Outcome: | The proposed tool beats baselines in citation metrics and personalized action-following with an online version of MySQA. |
MoDS: Moderating a Mixture of Document Speakers to Summarize Debatable Queries in Document Collections (2025.naacl-long)
Copied to clipboard
Nishant Balepur, Alexa Siu, Nedim Lipka, Franck Dernoncourt, Tong Sun, Jordan Lee Boyd-Graber, Puneet Mathur
| Challenge: | Query-focused summarization (QFS) gives an overview of documents to answer a query, ignoring debatable ones. |
| Approach: | They propose a multi-LLM framework that uses a Query-focused summarization approach to create balanced summaries that answer debatable queries. |
| Outcome: | The proposed framework beats SOTA by 38-59% in topic paragraph coverage and balance, based on new citation metrics. |
AI, Take the Wheel: What Drives Delegation and Trust in Human–Computer Cooperative Question Answering? (2026.findings-acl)
Copied to clipboard
Maharshi Gor, Yoo Yeon Sung, Yu Hou, Eve Fleisig, Zhu Irene Ying, Tianyi Zhou, Jordan Lee Boyd-Graber
| Challenge: | Human-AI collaboration is already happening, both in proactive delegation and deliberative adoption settings. |
| Approach: | They study delegating a task to AI without seeing its output and evaluating AI suggestions to decide whether to adopt them how AI output shapes final decisions. |
| Outcome: | The proposed game pairs 23 experts with 16 AI agents, capturing 387 delegation and 1440 adoption decisions. |
Personalized Help for Optimizing Low-Skilled Users’ Strategy (2025.naacl-short)
Copied to clipboard
Feng Gu, Wichayaporn Wongkamjan, Jordan Lee Boyd-Graber, Jonathan K. Kummerfeld, Denis Peskoff, Jonathan May
| Challenge: | a natural language agent generates moves and messages based on player intentions . a dozen games with novice and experienced players generate useful advice . |
| Approach: | a team of researchers augment a natural language agent to generate move and message advice . they use a game to simulate the intentions of novice and experienced players . |
| Outcome: | The enhanced agent generates move and message advice based on player intentions . the agent helps novices compete with experienced players and even surpass them . |
GRACE: A Granular Benchmark for Evaluating Model Calibration against Human Calibration (2025.acl-long)
Copied to clipboard
| Challenge: | Language models are often miscalibrated, leading to confidently incorrect answers. |
| Approach: | They propose a benchmark for language model calibration that incorporates comparison with human calibration. |
| Outcome: | The proposed metric analyzes model calibration errors and identifies types of miscalibration that differ from human behavior. |
A Good Plan is Hard to Find: Aligning Models with Preferences is Misaligned with What Helps Users (2025.emnlp-main)
Copied to clipboard
Nishant Balepur, Matthew Shu, Yoo Yeon Sung, Seraphina Goldfarb-Tarrant, Shi Feng, Fumeng Yang, Rachel Rudinger, Jordan Lee Boyd-Graber
| Challenge: | We test alignment methods to ensure LLMs are helpful, but they train or evaluate on what users prefer . |
| Approach: | They test alignment methods to ensure LLMs generate plans that help users . they get 4388 plan executions and 5584 comparisons to measure user preferences . |
| Outcome: | The proposed approach can be applied to the problem of user preferences and helpfulness. |
Large Language Models Are Effective Human Annotation Assistants, But Not Good Independent Annotators (2026.findings-acl)
Copied to clipboard
| Challenge: | State-of-the-art NLP models are expensive and inefficient for event annotation. |
| Approach: | They propose to integrate LLMs into a holistic workflow that summarizes news with event coreference resolution and argument extraction in three modes: AI-only, AI assistance, and human only. |
| Outcome: | The proposed workflow integrates LLMs to alleviate human labor in a holistic pipeline. |
Group Preference Alignment: Customizing LLM Responses from In-Situ Conversations Only When Needed (2025.emnlp-industry)
Copied to clipboard
Ishani Mondal, Jack W. Stokes, Sujay Kumar Jauhar, Longqi Yang, Mengting Wan, Xiaofeng Xu, Xia Song, Jordan Lee Boyd-Graber, Jennifer Neville
| Challenge: | Existing methods for group-aware adaptation capture divergent preferences from real-world conversation logs into interpretable rubrics. |
| Approach: | They propose a group-aware personalization framework that captures context-specific preferences and steers LLMs accordingly. |
| Outcome: | The proposed framework improves group alignment without compromising perfomance on benchmarks. |
BenchMarker: An Education-Inspired Toolkit for Highlighting Flaws in Multiple-Choice Benchmarks (2026.acl-long)
Copied to clipboard
Nishant Balepur, Bhavya Rajasekaran, Hyunjin Jane Oh, Michael Xie, Atrey Desai, Vipul Gupta, Steven James Moore, Eunsol Choi, Rachel Rudinger, Jordan Lee Boyd-Graber
| Challenge: | Multiple-choice question answering (MCQ) is standard in NLP, but benchmarks lack rigorous quality control. |
| Approach: | They propose an education-inspired toolkit that uses LLM judges to flag flaws in MCQs . they validate the tool with annotations and run it to audit 12 benchmarks based on 19-rule education rubric . |
| Outcome: | The proposed toolkit flags three common MCQ flaws based on a 19-rule education rubric . contaminated MCqs tend to inflate accuracy, while writing errors lower it and change rankings . |